building an efficient indexing for crawling the website with an efficient spider

with the present effort, we propose to investigate results of applying the right-truncated index-based web search engine in order to determine its usefulness for storing and retrieving arabic documents. the right-truncated index-based web search engine, being a program for reading any set of arabic documents accepts a query, and then processes both the documents and the query. thus, it selects (predicts) those documents most relevant to the query which has been inserted. the program encompasses both a morphological component and a mathematical one. the morphological component allows the researcher to run either a stemming algorithm or a right-truncated algorithm. the chief advantage of the stemming algorithm is that it uses the least possible amount of storage for indexing by mapping the inflected and derived terms into a single, indexed-stem word. on the other hand, the right-truncated algorithm reduces the amount of storage to a lesser degree, but increases the probability of retrieving relevant (user-favorable) documents, compared to the stemming algorithm. one of the purposes of our investigation is to compare the efficiency of these two indexing mechanisms. the mathematical component of the algorithm accepts the output of the right truncation algorithm, and then employs both term-frequency and inverse document-frequency (tf-idf) in order to establish the relative importance of each document, respective to the terms of the query. this paper also describes building a simple search engine based on a crawler or a spider. the clawer which indexes different types of documents is an algorithm to crawl the file systems from specified folder. a basic design and object model was developed to support single search word results as well as multiple search words results. it is capable of finding data to index by following (tracing) web links rather than searching directory listings in the file system. in this process files are downloaded through http and html pages parsed in order to obtain more links without getting into a recursive loop. also, this paper discusses how to improve indexing mechanism efficiency using a right truncated stemmer in terms of arabic documents processing.